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ABSTRACT 
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canonical correlation analysis (OVERALS) is discussed. Starting 
points are the delta method equations. The jackknife and bootstrap 
methods are compared for providing finite difference approximations 
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method indicates that the vector of profile proportions is perturbed 
by leaving out single observations. The grid of perturbed values is - 
used to estimate relevant derivatives. Bootstrapping means resampling 
with replacement from the original sample. Both procedures, 
bootstrapping and jackknifing, are used to compute pseudo-value means 
and standard errors for four different data sets: (1) the 
characteristics of 36 kinds of marine uammals; (2) data describing 
the attributes of 47 countries; (3) data from a study of school 
choice for 520 children leaving elementary school; and (4) a sample 
of 4,863 secondary students from the Second International Mathematics 
Study. For the small data sets the jackknife and bootstrap were used; 
for the larger sets, Monte Carlo versions of both were used. The 
jackknife method appeared less imprecise than the bootstrap method, 
and jackknife approximations were less stable for smaller samples. It 
is concluded that the bootstrap method performed better than did the 
jackknife method. For large samples, the bootstrap procedure works 
quite well for computing confidence intervals, and eigenvalues 
computed from OVERALS seem quite stable. Eight tables present the 
values from the fear data sets, and four figures illustrate category 
quantifications for all data sets. (Author/SLD) 
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ABSTRACT 



In this paper we discuss the estimation of mean and standard 
errors of the eigenvalues and category quantifications in gener- 
alized nonlinear canonical correlation analysis (OVERALS). Starting 
points are the delta method equations, but the jackknife and 
bootstrap are used to provide finite difference approximations to 
the derivatives. 

Keywords: canonical correlation analysis, delta method, 
jackknife, bootstrap, confidence interval, nonlinear 
transformation. 



\ 



ERLC 



4 



Jackknife and Bootstrap 
2 



INTRODUCTION 



Nonlinear canonical correlation analysis with k sets of vari- 
ables, OVERALS, Is a Multivariate technique in the sense of Glfi 
(1981, chap. 6). It 1s described by Oe Leeuw (1984a), Van der Burg. 
De Leeuw and Verdegaal (1984, 1986) and Verdegaal (1985). The k 
sets of variables are related in a linear way, as In ordinary 
canonical correlation analysis, but at the same time the variables 
are transformed nonllnearly. This can be formulated as a least 
squares problem minimizing the sum of squared deviations between 
unknown object scores and linear combinations of transformed vari- 
ables, organized in sets (Van der Burg et al., 1984, 1986) 

In current Implementations of OVERALS the variables are categ- 
orical, I.e. they assume only a small number of possible values. 
The technique assigns a numerical score to each category, the so- 
called category quantification. Scores are assigned In such a way 
that the sum of the t largest eigenvalues (generalized canonical 
correlations) Is maximized, while at the same time the measurement 
characteristics are respected. Thus for ordinal variables we Impose 
ordinal restrictions on the category quantifications, for numerical 
variables the category quantifications must even be linear with the 
original scores. For nominal variables there are no additional 
restrictions. 

The scoring system we just outlined gives a single quantifi- 
cation for each variable. It Is also possible to obtain multiple 
quantifications for each variable by using copies. (De Leeuw, 
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1984a; De Leeuw, 1985; Tljssen, 1985; Van der Burg et al., 1986). 
This «eans that a variable occurs more than once in a set. When 
only nominal characteristics are employed for all copies, the 
measurement level Is called multiple nominal. In case only one 
quantification Is dealt with the measurement level is called single 
nominal, single ordinal, or numerical. 

Because variables are categorical the profile for each Individ- 
ual only assumes a finite number of values. The quantities computed 
In canonical correlation analysis (category quantifications, object 
or profile scores, canonical correlations, correlations between 
quantified variables and canonical variables, and so on) are func- 
tions of the profile frequencies. 



THE DELTA METHOD 

We shall develop our statistical methods In a general multi- 
nomial context, not necessarily In terms of profile frequencies or 
proportions, and not directly applied to OVERALS output. The data 
are a vector p of proportions, based on a simple random sample of 
size n. Thus we suppose that p 1$ a realization of the random vari- 
able £, where n£ has a multinomial distribution with parameters 
(n,*). We Imbed the variable £ in a sequence j>n» /here nj^ Is also 
multinomial with parameters (n,it). For the expected value and the 
dispersion we have E(g n )«* and C(g n )»n" 1 (n-nn' ), where n «diagU). 
We also have convergence in law to a normal distribution, in the 
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sense that i n *n 1/2 (g n -x) + N(O.V). with V short for n-xn'. 

Now suppose ♦ Is a real valued function defined for all p (or 
all p close to *). and twice continuously differentiate at n. Then 
the delta nethod (Rao, 1965, section 6. a. 2, contains a nice dis- 
cussion) states that n 1/2 U(g n )-*(x)) ♦ N(O f g'Vg) f where g Is the 



vector of parti als of + at x. We can easily understand this result 

-1/2 

by writing as E n e * +n -n and then <level °P in 9 a Taylor series 
for ♦(g n ) In n" 1 ' 2 ^ around x. This gives: 

(1) ♦(g n ) - *(x) ♦ n'^V^ ♦ (2n)" 1 z n , Hz n + Optn" 1 ). 

A sequence of random variables Is Optn" 1 ) if nj^ converges In 
probability to zero. Matrix H contains the second order partials of 
* at *. The variance of ♦(g) Is given by 



(2) Var n U(g n )) « ^((♦(g n )) 2 ) - (E n U(g n ))) 2 . 



The second term of (2) is obtained by taking expectations of (1). 
As the random variable z,, converges in law to N(O.V) (the necessary 
conditions for E n (o p (n" 1 )) c ofn* 1 ) (cf. Serfling, 1980. section 
1.4) are satisfied) we get 

(3) E n ( * ( e n )) B * (n) + < 2n >" ltrHV ♦ o(n -1 ). 

The squared expectation Is 
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(4) < E *We n )))2 * UU))Z + "" l *(«HiW ♦ ofrf 1 ). 

The first term of (£) is obtained by squaring (1) and taking expec- 
tations: 

(5) E n (U( En )) 2 ) • U(*)) 2 ♦ n"^(n)trHV + n" l g'¥g < o(n -1 ). 
Subtraction of (5) and (4) gives 

(6) Var n L*(e n >) • n" : g ! Vg ♦ otn" 1 ), 

which corresponds to the delta method variance. Result (6) makes it 
possible to estimate the standard error of statistics of the form 
*(g n }. The estimate Is n~*gVg where g estimates g end V estimates 
V. Usually V-Vp-P-pp 1 with P*diag(p) f and g*g p the partlals of * at 
p. If we have an estimate a of the standard e-ror, then we also 
have an approximate confidence Interval (9SX) of the form 
U(E n M.96o, ♦(g n )+1.96o). 

We can also evaluate the bias of <Mg n ) as an estimate of 4>(n). 
To do so, expression (3) is used: 

(7) E^Uf^Wdi)) • (2n) -1 trHV ♦ otn" 1 ), 

so that estimates of H and V (e.g. H p and V p ) give an estimate of 
♦(h) with bias of order n~*- 

In many cases, and for instance In generalized canonical corre- 
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latlon analysis, the quantities of interest are defined by very 
complicated implicit functions. This makes It extremely tedious to 
compute first order derivatives, while second order derivatives are 
usually well nigh Impossible to obtain. In some special cases, such 
as correspondence analysis (Gifl, 1961, chap. 12; De Leeuw, 1984b; 
Schrlever, 1985) the delta method can be applied, but in other 
cases it simply Is not feasible. In such cases we can use a resam- 
pling method such as bootstrap and jackknlfe (Efron (1982) gives an 
overview), which we view here as methods to approximate the 
relevant partial s. Gifi (1981, cnap. 13) concentrates on the 
bootstrap. In this paper we use both jackknlfe and hootstrap and 
compare the results. 



THE MULTINOMIAL JACKKNIFE 

Suppose we drop one observation from the sample. If it has pro- 
rile number k, then the vector of profile proportions changes to 



(8) p(k) « p ♦ (n-l) sl (p-e k ). 



with e k the k th unit vector. The jackknife value for the observa- 
tion Is ♦(p(k)), and the pseudc value o k (p) is defined as 



(9) ^(p) * n*(p) - (n-lWp(k)). 
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The jackknife method uses the average pseudo value Ip^fp) a* an 
estimate of ♦ («) (p k are the elements of vector p), and uses the 
variance of the pseudo values as an estimate of g'Vg. We shall 
explain why this Is a reasonable procedure. Observe for the moment 
that no derivatives need to be computed. 

2 

Write for the average of the pseudo values, and o§ for their 
variance. Then combination of (8) and (9) using a Taylor series 
for ♦(p(k)) in (p-e k ) around p gives 

do) * k (p) = ♦(p) - g p '(p-e k ) - tfn-er^p-e^rH^p-e^ 
+ oUnr 1 ). 

Thus 

(11) * $ (p) • ♦(p) - tfr.-Zf^rVpHp + oUn)" 1 ). 

If we combine this result with the bias estimate provided by the 
delta method, given in (7), we see that 

(12) (n-l)(E n (* $ (e n )) - - 0. 

Thus, the average pseudo value corrects for bias, in the sar«e way 

as the delta method adjustment requiring second order derivatives. 
2 

To compete oj we observe that 

(13) * k (p) - ♦j(p) » -gMp-e^) + o(l). 
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From (13) It follows that 

(M) o\{p) * g p .V p g p . 
2 

Thus oj/n Is asymptotically equal to the delta method variance 
estimate. More precise expansions can be found in De leeuw (1985). 

The only computations that are required Is that the analysis 
technique Is repeated for each observation that Is successively 
dropped. For each jackknlfe sample a pseudo value Is computed. This 
gives new estimates of the relevant quantities. Their mean Is the 
Improved estimate, their variance Is an estimate of the stability. 
If the number of profiles Is much smaller than the number of obser- 
vations, then It Is more efficient to organize the computations In 
terms of profiles (as above), because repeating exactly the same 
analysis for observations with the same profile Is avoided. 



THE RANDOM JACKKNIFE 

Because generalized canonical correlation analysis with a large 
number of observations (and/or profiles) can be very expensive, It 
is often not feasible to compute all pseudo values. Instead we can 
estimate the average pseudo value and the variance of the p<eudo 
values by Monte Carlo methods. This amounts to leaving out one 
observation at random, and repeating this a number of times. If the 
sampling Is repeated this obviously converges to the theoretical 
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jackknife. 

It must be remarked that the delta method provides us with an 
approximation to the standard error. The jackknife in our Inter- 
pretation, gives an approximation to the delta method approximation 
of the standard error. And the Monte Carlo method approximates the 
jackknife approximations. Thus there are three levels of approxima- 
tion Involved. It does not follow, of course, that approximation of 
the true standard error becomes progressively worse, because there 
can be complicated Interactions between the three approximation 
processes. 

We also emphasize that 1t Is not necessary to present the jack- 
knife as an approximation of the delta method. It can also be 
Interpreted In Its own right as a method to stu4y stability, Indeed 
the Idea of Investigating the effect of 'leaving-one-out* also 
makes sense in a nonstochastlc context. 



THE MULTINOMIAL BOOTSTRAP 

If we look at the basic properties of the jackknife, as we have 
presented 1t, we see that the vector of profile proportions is 
perturbed by leaving out single observations. We apply our tech- 
nique to all these perturbed vectors, which are located reguarly 
around the observed vector, and we use this grid of perturbed 
values to estimate the relevant derivatives In a clever way. 
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Because observations are assumed to be equally important, each 
perturbation of the sample gets the same weight. I.e. occurs once 
in the distribution of all possible jackknife sample f we use 
the corresponding perturbations if the profile proportions we need 
weights Pj,...P m for the different profiles. 

The bootstrap is based on a different set of perturbations. In 
fact bootstrapping means resampling with replacement from the orig- 
inal sample, taking n observations (Efron, 1982). It means that we 
look at all vectors of profile frequencies adding up to n. It also 
means that the perturbations of the profile frequencies are 
centered around the sample value p, and that t. „ have weights 
according to tneir similarity with the sample value. Suppose 

(15) q * (nj/n n m /n) 

is a bootstrap perturbation of the profile proportions. Then tiie 
probability to occur in the distribution of all possible bootstrap 
samples 4 s 




n! 




P. 



The bootstrap pseudo values are defined as 



(17) 



♦Jp) " 2*(p) - ♦(q). 



The average bootstrap pseudo value is 
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(18) ♦j(p) • I q " q (P>* q (p>. 
and the bootstrap variance Is 



(19) o\{n) « I q " q (p)U q (p) - 0 S (P)) 2 . 



Substitute q s n~l/? z +p i n the pseudo values (17) and develop a 
Taylor series In vT l ^i around p. This gives 



(20) * q (p) ■ *(p> - n -I/2 g p 'z - UnfVy + o(n -1 ] 



Thus the average pseudo value Is 



(21) * S (P) « I" q (PU q (P) * #(P) - <2n) -1 trH p V p + otn" 1 ), 



as Iw q (p)z-0 and £w q (p)22'«V p . Combining (21) with (7) shows that 
the expected value of *£<£„) Is 

(22) Ms^n* " E n ( ^ ( En ) * IZnJ^trHV + o p (n -1 )). 
Combining (22) with (3) gives 

(23) n(E K ^5 ( E n ) " ♦ (m)) * °* 

Thus the average pseudo value Is an estimator of the population 
mean ♦(*) with bias of order if 1 - F © r the variance of the pseudo 
values we subtract (21) from (20) which gives 
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(24) * q (p) - * $ (p) - -«* 1/2 g p '2 - (2n)" 1 2 , Hpz 

+ (2n)* l trH p V p + o(n _1 ). 

Then the pseudo value variance Is 

(25) o|(p) - ""VVp + o("" 1 )- 

2 

The expected value of oj(B n ) converges to the delta method variance 

(26) n(E lt 0 $ ( t2n ,) * S'Vg. 

Thus the bootstrap variance Is an asymptotically unbiased estimator 
of n'lg'Vg. This result means that the variance of the bootstrap 
values (and the pseudo values) estimates the delta method variance. 
The jackknlfe pseudo value variance estimates g'Yg, so that the 
estimates of bootstrap and jackknlfe differ In a factor n. 

For large n a random version of the lackknlfe Is necessary. For 
the bootstrap a random version Is nearly always obligatory, as 
there are n n bootstrap samples possible. If the number of bootstrap 
samples is R, we need of course to take R large enough. Not very 
much research Is done on what Is large enough. Only Borsboom t Van 
Pelt (unpolished) did some research on this subject with regard to 
a computer program for nonlinear canonical correlation analysis 
(CANALS, cr. Van der Burg ft De Leeuw, 1983). They took bootstrap 
samples (ot 4241 observations) adding one at a time and recomputing 
standard errors of category quantifications. When the difference 
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between previous and current standard errors was smaller than .01 
they stopped taking bootstrap samples. Using the category with the 
largest standard error as a criterion, they judged that about 40 
samples Is enough for CANALS In this example. Their method Is Im- 
plemented In a computer program by Borsboom * Vlsser (1987). We 
repeated the analysis of Borsboom 6 Van Pelt with the Whales data 
(for a description see further on). We added 10 samples each time, 
starting at 10 and ending with 120. 



EXAMPLES 



In this section the jackknlfe and bootstrap procedures re used 
to compute pseudo value means and standard errors for four differ- 
ent data sets. For the smaller data sets (I.e. Wnales and Russett) 
a complete jackknlfe and a random bootstrap procedure were applied, 
while In the case of the FYTY and SIMS data sets a Monte Carlo 
version of jackknlfe and bootstrap were used. For all data sets the 
generalized canonical correlations (I.e. eigenvalues) were em- 
ployed. In addition 'or the Russett and FYTY data some category 
quantifications were also considered. An overview of the results of 
these analyses Is presented In tables found In the following sec- 
tions of this paper. Many of these tables have a jlumn for eigen- 
values (respectlvlly category quantifications), computed from the 
original data matrix (or sample), In single precision (SP) and 
double precision (DP). For single precision the convergence of the 
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OVERALS program Is computed with accuracy to 10% while for double 
precision It Is computed with accuracy to 10" 8 . The OVERALS conver- 
gence criterion Is specified In terms of the difference between the 
sum of eigenvalues for two consecutive Iterations In the alterna- 
ting least squares procedure. Many tables which follow have three 
columns for estimated population means, of which there are two for 
jackknlfe (single and double precision, JSP and JDP) and one for 
bootstrap (single precision, BSP). In addition many tables also 
contain estimated standard errors corresponding to the estimated 
means. 

The first example considered (the Whales data set, Vescla, 1985) 
consists of fifteen variables decrlblng characteristics of 36 
whales, porpoises and dolphins (e.g. form of the head, kind of 
feeding, place of blow hole, colour, etc.). Uslnq the twelve vari- 
ables without missing scores we repeated an analysis described by 
Van der Burg (1985). This means that the program OVERALS was used 
for homogeneity analysis (multiple correspondence analysis): twelve 
sets each consisting of one variable treated as multiple nominal. 

The second example considered was based on the Russett data set 
(Russett, 1964). These data contain three sets of va iables dealing 
with attributes of 47 countries. The first set contains two vari- 
ables concerning ownership of land, the second set contains the 
variables gross national product and percentage of people working 
In agriculture, while the third set Includes the four variables 
that are Indicators of political Instability. The multlchotomlzed 
scores originally established by Glfl (1981, chap. 7) were used In 
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this study. The data were analyzed with numerical, single nominal , 
and multiple nominal options. 

The third example (designated FYTY for From Year to Year) Is 
based on a sample from a large school career survey (for references 
see Oe Leeuw A Stoop, 1979). In the example we consider, there are 
520 school children and six variables divided Into three subsets. 
The variables are choice of school after primary education (set 1), 
achievement test score and teacher's advancement recommendation 
(set 2), educational level of father and mother, and profession of 
father (set 3). The different sets were measured at different 
points In time period, the order considered was: subset 3, subset 
2, subset 1. The FYTY data were analysed In the same three ways 
used with the Russet t data. 

The fourth example also comes from a school survey, 1n this case 
the SIMS (Second International Mathematics Study) Project (Pelgium, 
Eggen & Plomp, 1984 and 1986). For this example the complete data 
base (4863 school children) as well as a sample from that base 
(1000 school children) were considered. The eigenvalues correspond- 
ing to the complete data base were treated as population par- 
ameters, while the sample data were used to estimate these popu- 
lation parameters. The variables considered In this example were 
divided Into three subsets. One subset contains the variables type 
of school and father's education, another subset contains three 
attitudes towards mathematics, while the remaining subset Includes 
a mathematics test score. Four measurement levels were considered 
for the SIMS data example. Single nominal, ordinal and numerical, 
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and multiple nominal . 



RESULTS FOR WHALES 



Table 1 contains bootstrap estimates for the Whales example. As 
earlier specified, the bootstrap procedure was Implemented with 
various numbers of samples with Increments of size 10. The number 
of samples ranged from 10 to 120. Results for this example were 
computed In both single and double precision. The estimated popu- 
lation means and standard errors are presented In Table 1. It may 
be noted that the bootstrap pseudo value means show only minor 
differences, even for the extreme cases (I.e. number of samples of 
10 and 120). However, the estimated standard error tended to In- 
crease for number of samples 10 to 40, while remaining rather 
stable for number of samples greater than 40. In addition, It was 
found that differences were negligible between the two levels of 
precision. For the remaining examples 40 samples were considered 
when a random process was used In Identifying samples. In addition, 
only single precision computation was used with the bootstrap. 



INSERT TABLE 1 ABOUT HERE 



The eigenvalues computed from the Whales data are presented in 
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Table 2. Four methods were considered: SP, DP, API and CA. The 
first two use the Fortran OVERALS program, and result In similar 
eigenvalues. The third method Is based on an APL version of the 
OVERALS algorithm. Since APL Is quite precise (16 digits accuracy) 
the results obtained under this method may be considerably more 
accurate than results from the Fortran program. The rourlh method 
considered was correspondence analysis (CA). In those cases In 
which all variables are treated as multiple nominal, the OVERALS 
problem may be reformulated In a format which Is insistent with 
correspondence analysis. For this reason a correspondence analysis 
program (ANACOR, Gifl, 1985) was used. The eigenvalues from ANACOR 



INSERT TABLE 2 ABOUT HERE 



are precisely the same as the OVERALS eigenvalues. However the 
Variances do differ (see Table 3). Although the APL eigenvalues are 
slightly different from those obtained under the above mentioned 
approaches, those differences hardly seem worth mentioning. The 
Jackknife and bootstrap results are presented In column 4 to 11 of 
Table 2. The bootstrap results are hr 3e d on 120 samples, while the 
JSP and JDP results are based on 36 samples (a complete jackknife). 
The estimated means do not substantially differ from each other, 
but the estimated standard errors do manifest sizable differences. 
The three jackknife estlma. are very similar, however the boot- 
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strap estimates are considerably lower in magnitude than the jack- 
knife estimates. It Is clear from Table 1 that using more samples 
will not change the estimated standard error. Thus In this case the 
bootstrap results converge to values different from those of the 
jackknlfe method. As the number of observations Is very small In 
this example (I.e. 36) this m^y be due to the fact that asymptotic 
characteristics are not satisfied, and thus approximations may be 
Imprecise. We can compare the result? obtained under OVERALS with 
those obtained under CA. As It Is possible to compute the first 
order derivatives In case of correspondence analysis 9 the ANACOR 
program delivers variances (Table 3). The first two bootstrap 
estimates are sytematlcally lower than the CA values , while the 
corresponding jackknlfe estimates are higher. Only the third 
variance Is similar for the bootstrap and jackknlfe (and lower than 



INSERT TABLE 3 ABOUT HERE 



for CA). Assuming that CA gives the more precise value (direct 
computation Instead of approximation), we find that the jackknlfe 
overestimates and the bootstrap underestimates the standard error. 
We do not make conclusions on the third variance is the smallest 
eigenvalue normally Is much less precise than the larger ones. 
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RESULTS FOR RUSSETT 



A complete jackknlfe and a random bootstrap were applied to the 
Russett data. As the number of observations (47) Is still rather 
small we may expect different results from the bootstrap and Jack- 
knife analyses. The results for the eigenvalues are presented in 
Table 4. One of the most striking aspects of these results Is that 
the standard errors are considerably larger than those obtained 
with the Whales data. It should be realized however, that hom- 
ogeneity analysis Is known to be a very stable technique, and most 
likely an example with more variables In the sets will be less 
stable. JSP means appear to be rather strange, especially If we 
keep In mind that the data are not a sample but the population. JDP 
estimates seem to be a little less unexpected. Estimation by boot- 
strap Is more like the eigenvalues based on the complete data set 
than the estimation by jackknlfe. But In any case the results 
appear discouraging. Bootstrap standard errors are smaller In gen- 
eral than jackknife standard errors but they are too large for 
reasonable confidence Intervals. 



INSERT TABLE 4 ABOUT HERE 



A look at results for category quantifications of two variables 
(LABO and DEMO) Is even more perplexing (Table 5). The bootstrap 
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estimates provide at least a rough approximation of the corre- 
sponding population results. However the jackknlfe estimates are 
quite different from the population values. In addition, the corre- 
sponding standard errors seem extrordlnarlly large. A direct look 
at the bootstrap and jackknlfe results (without computing pseudo 
values) shows their values to be rather stable (Figures 1 and 2). 
Every dash (-) corresponds to a category quantification resulting 
from one bootstrap/ jackknlfe sample (called bootstrap/ jackknlfe 
sample value), while ^the symbol 0 represents the original category 
quantifications of the Russett data. On the horizontal axis the 
original category scores are rxaled. while on the vertical axis the 
category quantifications are scaled. As the jackknlfe and bootstrap 



INSERT FIGURES 1 AND 2 ABOUT HERE 



sample value variances differ asymptotically by a factor n. we 
divided the bootstrap sample values (In deviation from their mean) 
by n*^ 2 to make the results comparable for the two techniques. It 
may be noted that DEMO Is a much more stable variable than Is LABO. 
For the latter variable, cateories 1 A 2. and 4 & 5 respectively 
show considerable overlap. However, overall the results appear 
rather stable. 

Because the results for the Whales and the Russett examples wer? 
disappointing, additional examples Involving larger data sets were 
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considered* 



RESULTS FOR FYTY 



The FYTY data were analyzed by the Monte Carlo version of jack- 
knife and bootstrap. The resulting eigenvalue estimates are pre- 
sented In Table 6. Considering the FYTY data as a sample from a 
large data base, we can expect eigenvalues obtained from the orig- 
inal sample to be larger than estimates obtained from bootstrap and 
jackknlfe. This Is because sample estimates tend to overestimate 
the population parameters while the jackknlfe and bootstrap pseudo 
value means are supposed to reduce the bias found In sample esti- 
mates. The actual estimation outcomes that were obtained for the 
FYTY data are as follows. The jackknlfe estimates were rather simi- 
lar to one other but do not seem to show any relation with the 
sample eigenvalues. Here the JSP estimation seems to have outper- 
formed the JDP estimation, but that Is not really true. The JSP 
values have been produced with three digits accuracy, which was not 
sufficiently precise. For Instance the second estimate of the popu- 
lation mean (multiple nominal), seems very good for JSP, and some- 
what worse for JDP. This difference Is due to the number of digits 
provided In the computer output. As all jackknlfe values differed 
from the sample value In the fourth and higher digits, we obtain 
the same values when we truncate at three digits. Consequently, 
sample and jackknlfe estimates are the same. The JDP estimate, In 
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which 6 digits accuracy Is provided. Is much larger, but must be 
Bore precise. Thus In this case the JSP results may be somewhat 
suspect. For this reason we look only at the JDP results. For the 
following example we used output with 6 digits accuracy to avoid 
this problem. 



INSERT TABLE 6 ABOUT HERE 



The bootstrap estimates of the population mean appear better 
than the jackknlfe estimates. The bootstrap estimates are Indeed 
always smaller than the sample eigenvalues. For all measurement 
levels the first value provided a closer approximation than the 
second value. Many eigenvalue routines give more precise results 
for the larger eigenvalues (supposing that the smaller ones are 
also computed). This phenomenon may provide an explanation for the 
difference in accuracy obtained for first and second eigenvalues. 

Due to the large number of observations stardard errors are 
smaller than found for the Russett example. However especially In 
the case of jackknlfe these standard errors are itlll unacceptably 
large. 

Until 1 now It would seem that the bootstrap method is preferable 
to the Jackknlfe method, since It apprears to provide more precise 
estimates with smaller variance. As the number of sample observa- 
tions Is still not large, an additional sample size which more 
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closely approximates the asymptotic case seemed warrented. For this 
reason the next example Is based on 1000 observations. 
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However, before this next example Is presented some category 
quantifications of the FYTY data are considered. To do this, the 
variables PRE and TON are used (Table 7). Looking at the standard 
errors for these variables* It may be noted that TON Is much more 
stable than PRE. We also see that the standard errors of the cate- 
gory quantifications are larger than those for the eigenvalues 
(Table 6). As category quantifications are proportional to the 
square root of eigenvalues, their standard errors Mill Increase at 
a corresponding rate. In Table 7 we see that again the JSP and JOP 
results agree rather well, but the correspondence between sample 
eigenvalues and the estimated population means Is almost nonexis- 
tent for the jackknlfe. Bootstrap result* are very similar tc the 
sample values for TON, however results for ^E show iess similarity 
(but not as bad as jackknlfe estimates). Looking at the category 
quantifications the jackknlfe and bootstrap samples directly 
(like we did for the Russet example), we find that those variables 
behave very stably (Figures 3 and 4). In the case of TON all fourty 
dashes (jackknlfe /bootstrap sample values) fall together both for 
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INSERT FIGURES 3 AND 4 ABOUT HERE 



Jackknlfe and bootstrap, while for PRE we find some variability 
amoung the category quantifications. Note that In case of JDP, most 
of the variability that Is present Is due to a single outlying 
sample. This stability In the results Is very striking, 

RESULTS FOR SIMS 

In Table 8 we find results for the eigenvalues of the SIMS data. 
The data base Itself consisted of 4863 observations and served as 
the poputation of Interest. For the purpose of estimating the popu- 
lation eigenvalues a sample of 1000 observations was considered. 
The bootstrap and jackknlfe approaches were used (on the sample) 
for reestlmatlon of the population values (with the computer re- 
sults specified to 6 digits for both SP and DP). Note that boot- 
strap estimates are always between the population and the sample 
values. Theoretically this Is to be expected The same outcome Is 
also expected when the jackknlfe Is used. However* this did not 
occur In half the cases of Table 8. Since standard errors appear to 
be reasonably small, 1t Is suggested that confidence Intervals for 
bootstrap results be computed. 
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DISCUSSION 

In comparing jackknife with bootstrap one Is left with the im- 
prest on t* " the jackknife method Is far more Imprecise than the 
bootstrap method. Apparently jackknife approximations are less 
stable for smaller samples. The use of double precision computa- 
tions tends to Improve the jackknife results, but does not elim- 
inate the problem. For the bootstrap method it does not matter 
whether computations are SP or DP. 

For smaller samples the jackknife standard error is larger than 
the bootstrap standard error. It is possible that approximations 
are not precise enough because of the small sample size. It appears 
that the jackknife and bootstrap method converge to different 
values. 

Jackknife results are sensitive to precision and thus to compu- 
tational error. This can be concluded from the differences in SP 
and DP results. This finding does not vary with sample size. It 
concerns mainly the estimated population means of eigenvalues and 
not the corresponding standard errors. Estimation based on boot- 
strap is more robust than estimation based on jackknife. Results 
for category quantifications are discussed further on. 

For smaller samples the estimated standard errors of eigenvalues 
(both for jackknife and bootstrap) are unacceptably large (varying 
from .025 to 1.05). Such error variation would result in huge con- 
fidence intervals (with substantial overlap among Intervals). 

In all cases considered, the estimated population means for 



ERLC 



28 



JackknUe and Bootstrap 
26 



eigenvalues appear to be worse for Jackknlfe than for bootstrap, as 
bootstrap estimates are nearer to the population or sample valu*. 
than jackknlfe estimates are. Even for the largest sample con- 
sidered, more deviations (I.e. population mean not In 95% confi- 
dence Interval) were found with jackknlfe than with bootstrap. 
Deviations occur for r.econd eigenvalues (ordinal and numerical) of 
JDP and for all second eigenvalues of JSP (Table 8). Many eigen- 
value routines are less precise for the smallest computed eigen- 
value. As jackknlfe 1s sensitive to precision this may be the 
cause. 

Estimates for category quantifications seem rather strange. 
Category quantifications correspond to the square root of eigen- 
values, and thus will be less precise. However the estimates are 
sometimes highly Inaccurate, even for very stable jackknlfe or 
bootstrap sample values. Again 1n this case the bootstrap seems to 
perform much better than the jackknlfe (Tnble 7). In several plots 
stability of results Is Illustrated (Figures 1 to 4). Only In the 
case of LABO are the results unstable. This variable has a much 
smaller weight than the other var ables used for Illustration. It 
may be that this smaller weight results in a lower level of stab- 
ility. 

In the case of the example based on the largest sample, the 
bootstrap means computed directly from bootstrap samples provided a 
less accurate estimate of the population value than was provided by 
pseudo value means which are presented In Table 8. A similar out- 
come was found (although the differences were less) for jackknlfe. 
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Using pseudo values tends to provide a better approximation than 
obtained Ly averaging bootstrap and jackknlfe results directly* 

If we compare other studies using jackknlfe and bootstrap, we 
find that Boomsiu (1986) concludes that both methods give very 
similar results for a sample size of 100. He estimates parameters 
for Covarlance Structure Analysis. We cannot confirm his findings 
In the case of 0YERAL3 parameters. 

In conclusion we can say that the bootstrap method performed 
better than the jackknlfe method. For larger samples the bootstrap 
procedure works quite well for computing confidence Intervals. The 
use of 40 samples seems to be sufficient for estimation, but was 
not thoroughly Investigated In our study. For larger samples eigen- 
values computed from OVERALS seem quite stable. However, category 
quantifications seem to result In much wider confidence Intervals. 
Studying results for category quantifications directly (I.e. jack- 
knife and bootstrap sample values) leads to more positive con- 
clusions which are more In agreement with other's earlier experi- 
ence with the examples here considered (cf De Leeuw & Stoop, 1979; 
Glfl, 1981, chap. 7). 
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TABLE 1 



WHALES, muUlple nominal. Estimated population means and es- 
timated standard errors for bootstraps with different sample 
sizes. SP-slngle precision, DP»dout>1e precision. 



Whiles pop. mean st. error pop. mean st. error 

Bootstr SP DP S? DP Bootstr SP DP SP DP 



10 


.601 
.397 
.306 


.601 
.398 
.305 


.031 
.058 
.022 


.031 
.058 
.017 


70 


.608 
.395 
.298 


.608 
.396 
.299 


.042 
.041 
.026 


.042 
.041 
.025 


20 


.602 
.393 
.308 


.602 
.393 
.308 


.029 
.049 
.020 


.029 
.049 
.018 


80 


.608 
.394 
.297 


.607 
.394 
.298 


.043 
.044 
.025 


.043 
.044 
.025 


30 


.605 
.395 
.302 


.605 
.396 
.303 


.032 
.046 
.024 


.032 
.046 
.023 


90 


.610 
.394 
.297 




.045 
.046 
.025 




40 


.606 
.391 
.298 


.606 
.392 
.299 


.041 
.043 
.027 


.041 
.043 
.026 


100 


.612 
.394 
.296 




.047 
.044 
.025 




50 


.609 
.391 
.299 


.609 
.392 
.300 


.040 
.042 
.027 


.040 
.042 
.026 


no 


.612 
.393 
.295 




.046 
.043 
.025 




60 


.610 
.394 
.298 


.610 
.395 
.299 


.039 
.041 
.027 


.039 
.041 
.026 


120 


.613 
.393 
.295 




.045 
.042 
.026 
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TABLE 2 



WHALES, wltlple nominal. Eigenvalues, estimated population 
Means and estimated standard errors for jackknlfe (J) (36 
samples), bootstrap IB) (120 samples) and OVERALS-APL. SP* 
single precision, DP'double precision. 



Whiles eigenvalue population mean standard error 

DP 1 APL JSP JDP JAPL BSP JSP JDP JAPL BSP 



MULT .635 .637 .618 .611 .614 .613 .056 .056 .056 .045 
MOM .413 .415 .385 .398 .400 .393 .060 .060 .061 .042 
.317 .320 .279 .282 .285 .295 .028 .028 .026 .026 



Eigenvalues SP=DP»CA (Correspondence Analysis) 
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TABLE 3 



WHALES, estimated variances for Jackknlfe 
(36 samples), bootstraps (120 samples), 
and correspondence analysis. SP»s1ngle 
precision, DP-double precision. 



Whales 


JSP 


JOP 


BSP 


CA 


estimated 


.0032 


.0031 


.0020 


.0025 


variances 


.0037 


.0036 


.0018 


.0031 




.0008 


.0008 


.0007 


.0012 
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TABLE 4 



RUSSETT, three measurement levels. Eigenvalues, estimated 
population means and estimated standard errors. SP«single 
precision, DP»double precision, J»jackkn1fe (47 samples), 
B«bootstrap (40 samples). 



Russett eigenvalue population mean stand, error 
SP DP JSP JDP BSP JSP JDP BSP 



single 
nominal 


.765 
.710 


.770 
.706 


.420 
.924 


.703 
.616 


.673 
.638 


.065 
.105 


.058 
.089 


.032 
.034 


numerical 


.687 
.461 


.687 
.463 


.674 
.479 


.556 
.412 


.660 
.400 


.057 
.095 


.057 
.076 


.048 
.047 


Multiple 
nominal 


.815 
.736 


.815 
.734 


.710 
.450 


.690 
.443 


.732 
.625 


.049 
.053 


.049 
.054 


.029 
.025 
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TABLE 5 



RUSSETT, single nominal. Frequencies* category quantifications, 
estimated population means and estimated standard errors for two 
variables* SP»s1ngle precision, DP*doub1e precision, J»jacicicn1fe 
(47 samples), B'bootstrap (40 samples). 



Russett cat. quant. population mean standard error 

freq. SP DP JSP JDP BSP JSP JDP BSP 



LABO 


















10 


-1.21? 


-1.157 


-3.226 


-.556 


-1.240 


.969 


.835 


.699 


11 


-.970 


-1.019 


.229 


-1.969 


-1.191 


.931 


.805 


.476 


14 


.622 


.606 


1.120 


.465 


.662 


.493 


.445 


.455 


11 


1.176 


1.200 


1.028 


1.831 


1.410 


.706 


.587 


.392 


DEMO 


















1 


1.147 


1.106 


1.434 1 


0.5551 


1.0822 


1.3591 


.9941 


.677 2 


19 


-1.090 


-1.085 


-.674 


-.480 


-1.049 


.322 


.294 


.305 


12 


-.400 


-.410 


-1.272 


-1.610 


-.553 


.614 


.529 


.544 


19 


1.171 


1.173 


1.406 


1.448 


1.284 


.215 


.207 


.261 



519 observed values 
511 observed values 
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TABLE 6 



FYTY, three measurement levels. Eigenvalues, estimated popu- 
lation weans and estimated standard errors. SP* single preci- 
sion, DP-double precision, J«jackknife, B»bootstrap (both 40 
samples). 



FYTY eigenvalue population mean stand, error 

SP DP JSP JDP BSP JSP JDP BSP 



single 
nominal 


.735 
.399 


.734 
.400 


1.202 
.269 


1.060 
.167 


.731 
.367 


.061 
.034 


.056 
.025 


.021 
.017 


numerical 


.695 
.341 


.695 
.341 


.851 
.639 


.679 
.567 


.692 
.326 


.016 
.057 


.014 
.054 


.017 
.013 


multiple 
nominal 


.742 
.555 


.742 
.555 


.690 
.555 


.695 
.626 


.729 
.535 


.014 
.019 


.012 
.026 


.018 
.016 
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TABLE 7 



FYTY, single nominal. Frequencies, category quantifications, 
estimated population means and estimated standard errors tor 
two variables. SP*s1ngle precision, DP*dout>1e precision, 
J'jacklcnlfe, B*bootstrap (both 40 samples). 



FYTY 


cat. quant 


population mean 


standard error 


freq. 


SP 


DP 


JSP 


JDP 


BSP 


JSP 


JDP 


BSP 


TON 


















42 


-.586 


-.586 


-2.052 


-2.065 


-.584 


.175 


.173 


.128 


208 


-.853 


-.853 


-1.774 


-1.748 


-.851 


.077 


.075 


.087 


165 


.056 


.056 


2.119 


2.132 


.080 


.125 


.122 


.146 


105 


1.853 


1.853 


.317 


.291 


1.867 


.125 


.124 


.125 


PRE 


















49 


-.622 


-.622 


-43.102 


-40.287 


-.617 


6.574 5.641 


.581 


147 


-.592 


-.592 


-23.000 


-20.275 


-.694 


4.157 3.696 


.457 


181 


-.184 


-.184 


8.98V 


7.886 


-.126 


i.587 2.143 


.295 


111 


.298 


.298 


33.994 


32.606 


.200 


4.147 3.598 


.676 


32 


3.680 


J.680 


5/795 


-.212 


4.275 


3.022 2.350 


.811 
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TABLE 8 



SIMS, four Measurement levels. Eigenvalues of popula- 
tion (POP) and sample (SAM), estimated population 
means and estimated standard errors, SP«s1ng1e preci- 
sion, DP«doub1e precision, J'jackknlfe, B*bootstrap 
(both 40 samples). 



SIMS eigenvalue population mean sMnd. error 

POP 1 SAM 1 JSP JDP BSP 2 JDP BSP 



multiple 
nominal 


.615 
.450 


.625 
.463 


.615 
.566 


.620 
.463 


.619 
.454 


.011 
.013 


.011 
.013 


.011 
.012 


single 
nominal 


.613 
.352 


.622 
.377 


.621 
.171 


.622 
.377 


.616 
.371 


.014 
.015 


.013 
.013 


.012 
.010 


single 
ordinal 


.613 
.352 


.622 
.377 


.728 
.602 


.628 
.697 


.619 
.375 


.014 
.094 


.014 
.092 


.012 
.012 


numerical 


.506 
.350 


.615 
.363 


.609 
.277 


.608 
.321 


.613 
.359 


.012 
.010 


.012 
.009 


.012 
.011 



1 No difference between eigenvalues SP and OP, except 
for the first eigenvalue of single nominal (SP,SAM): 
.621. c 

2 Bl)P with convergence to 10° gives the same results. 
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FIGURE CAPTIONS 



Figure 1. RUSSETT. Category quantifications for LABO based on three 
Methods: JSP, JDP and BSP (0 orglnal sample value, - jackknlfe/ 
bootstrap sample value). 

Figure 2. RUSSETT. Category quantifications for DEMO based on three 
methods: JSP, JDP and BSP (0 original sample value, - jackknlfe/ 
bootstrap sample value). 



Figure 3. FYTY. Category quantifications for TON based on three 
methods: JSP, JDP and BSP (0 original sample value, - jackknlfe/ 
bootstrap sample value). 



Figure 4. FYTY. Category quantifications for TON based on three 
methods: JSP, JDP and BSP (0 original sample value, - jackknlfe/ 
bootstrap sample value). 
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RUSSETT-JSP RUSSETT-JDP RUSSETT-BSP 
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FVTY-JSP FYTY-JOP FYTY-BSP 
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PIGURE 3 
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